Move repository-s3 fixture tests to QA third party project #29372

tlrx · 2018-04-04T13:53:26Z

This commit moves the repository-s3 fixture test added in #29296 in a new qa/third-party/amazon-s3 project. This new project allows the REST integration tests to be executed using the real S3 service when
all the required environment variables are provided. When no enviromnent variable is provided, the tests are executed using the fixture added in #29296.

The REST tests located in the plugin project now only verify that the plugin is correctly loaded.

The REST tests in the QA project have been adapted to allow a bucket name and a base path to be specified as env vars. This way it is possible to run the tests with different base paths (could be anything, like a CI job name or a branch name) without multiplicating buckets.

Related to #29349

elasticmachine · 2018-04-04T13:53:29Z

Pinging @elastic/es-distributed

tlrx · 2018-04-05T08:20:32Z

qa/third-party/amazon-s3/build.gradle

+}
+
+/** A task to start the AmazonS3Fixture which emulates a S3 service **/
+task s3Fixture(type: AntFixture) {


Note: the task is always created but not used when env vars are provided. I find it easier this way.

tlrx · 2018-04-17T09:48:25Z

@elastic/es-core-infra Can someone from the team can review this pull request please? Or maybe do you prefer to hold on this change until another major change is merged? (I'm fine with this but I'd like to know as I have few others PRs that are blocked by this one)

Note that this change should not impact the CI as it will use the fixture to run tests by default.

ywelsch

Recent discussions (Core/Infra meeting yesterday) have concluded that we should put QA projects that are really only testing a single plugin (and not the integration of many) into a subdirectory of that project and hence as a sub-project in Gradle. One of the biggest advantages is that this allows you to test all aspects related to that plugin with just one simple Gradle call. Examples for this are the CCR plugin (see https://github.com/elastic/elasticsearch/tree/ccr/x-pack/plugin/ccr) or the index lifecycle plugin.

The change looks good otherwise.

ywelsch · 2018-04-26T09:34:20Z

qa/third-party/amazon-s3/build.gradle

+    s3SecretKey = 's3_integration_test_secret_key'
+    s3Bucket = 'bucket_test'
+    s3BasePath = 'integration_test'
+    useFixture = true


I wonder if we should log something if the fixture is not used, i.e., using external service (bucket + path).

I don't think we should log the bucket name, instead it will print a simple "Using an external service to test the repository-s3 plugin" message

tlrx · 2018-04-26T12:37:35Z

Thanks @ywelsch, I didn't know about this decision. The QA test is now located in the plugins/repository-s3/qa/amazon-S3 project. I hooked up the check task so that executing :plugin:repository-s3:check also executes :plugin:repository-s3:qa:amazon-s3:check.

I had to adapt the project fileq of x-pack smoke tests so that they didn't try to include :plugin:repository-s3:qa:amazon-s3 as if it was a real plugin.

ywelsch

LGTM

ywelsch · 2018-04-26T13:33:39Z

x-pack/qa/smoke-test-plugins-ssl/build.gradle

-  // need to get a non-decorated project object, so must re-lookup the project by path
-  integTestCluster.plugin(subproj.path)
-  pluginsCount += 1
+  if (subproj.plugins.hasPlugin(PluginBuildPlugin) || subproj.plugins.hasPlugin(MetaPluginBuildPlugin)) {


I wonder if this is too much magic. An alternative might be to write project.project(':plugins').getChildProjects.each { n, subproj -> ... }
to just range over direct children of plugins.

Thanks. This is indeed a good suggestion. I'll update this.

rjernst · 2018-04-26T15:01:05Z

Recent discussions (Core/Infra meeting yesterday) have concluded that we should put QA projects that are really only testing a single plugin (and not the integration of many) into a subdirectory of that project and hence as a sub-project in Gradle.

While this was discussed, it was not concluded. Please don't do this until there is agreement on how (or if) it should be done.

jasontedor · 2018-04-26T21:06:06Z

While this was discussed, it was not concluded. Please don't do this until there is agreement on how (or if) it should be done.

We had consensus on the point of tests for features/modules/plugins being next to the feature/module/plugin as sub-projects (e.g., we mentioned examples of CCR/index lifecycle management/reindex). There are some follow-up discussions needed related to the standing up of clusters with various features enabled/disabled, and related to security, but on the broader point of co-locating qa-style tests with the code, we did have consensus in the interest of optimizing workflow for developers on these.

It is apparent that you feel strongly otherwise so we can re-discuss this. However, I do not want to block @tlrx on this discussion where he was acting on guidance from his reviewer who saw the outcome of our previous discussion similarly. I will approve this PR as-is and if we come to a different decision when we discuss this again we can move the tests back (and we will not burden @tlrx with this work), this is okay.

This will be on the agenda for our next core/infra meeting.

This commit moves the repository-s3 fixture test added in elastic#29296 in a new `qa/third-party/amazon-s3` project. This new project allows the REST integration tests to be executed using the real S3 service when all the required environment variables are provided. When no env var is provided, then the tests are executed using the fixture added in elastic#29296. The REST tests located in the plugin project now only verify that the plugin is correctly loaded. The REST tests have been adapted to allow a bucket name and a base path to be specified as env vars. This way it is possible to run the tests with different base paths (could be anything, like a CI job name or a branch name) without multiplicating buckets. Related to elastic#29349

tlrx · 2018-04-27T14:56:25Z

Thanks a lot @ywelsch @jasontedor @rjernst!

This pull request has been merged with the QA project being a sub-project of the repository-s3 plugin. If this decision is revisited by the core/infra team, please don't hesitate to reach me out - I'd be happy to help to move them again.

This commit moves the repository-s3 fixture test added in #29296 in a new `repository-s3/qa/amazon-s3` project. This new project allows the REST integration tests to be executed using the real S3 service when all the required environment variables are provided. When no env var is provided, then the tests are executed using the fixture added in #29296. The REST tests located at the `repository-s3`plugin project now only verify that the plugin is correctly loaded. The REST tests have been adapted to allow a bucket name and a base path to be specified as env vars. This way it is possible to run the tests with different base paths (could be anything, like a CI job name or a branch name) without multiplicating buckets. Related to #29349

* master: (7173 commits) Bump changelog version to 6.4 (elastic#30217) [DOCS] Adds native realm security settings (elastic#30186) Test: Switch painless test to 1 shard CCS: Drop http address from remote cluster info (elastic#29568) Reindex: Fold "from old" tests into reindex module (elastic#30142) Convert FieldCapabilitiesResponse to a ToXContentObject. (elastic#30182) [DOCS] Added 'on a single shard' to description of max_thread_count. Closes 28518 (elastic#29686) [TEST] Redirect links to new locations (elastic#30179) Move repository-s3 fixture tests to QA test project (elastic#29372) Fail snapshot operations early on repository corruption (elastic#30140) Docs: Document `failures` on reindex and friends Build global ordinals terms bucket from matching ordinals (elastic#30166) Watcher: Ensure mail message ids are unique per watch action (elastic#30112) REST: Remove GET support for clear cache indices (elastic#29525) SQL: Correct error message (elastic#30138) Require acknowledgement to start_trial license (elastic#30135) Fix a bug in FieldCapabilitiesRequest#equals and hashCode. (elastic#30181) SQL: Add BinaryMathProcessor to named writeables list (elastic#30127) Tests: Use buildDir as base for generated-resources (elastic#30191) Fix SliceBuilderTests#testRandom failures ...

tlrx added >test Issues or PRs that are addressing/adding tests review :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v7.0.0 labels Apr 4, 2018

tlrx requested a review from rjernst April 4, 2018 13:53

tlrx mentioned this pull request Apr 4, 2018

Test cloud based repository plugins using real services backends #29349

Closed

3 tasks

tlrx commented Apr 5, 2018

View reviewed changes

tlrx force-pushed the add-qa-third-party-amazon-s3 branch 2 times, most recently from c5aca22 to 9979f1e Compare April 17, 2018 07:47

tlrx force-pushed the add-qa-third-party-amazon-s3 branch 3 times, most recently from f945030 to 914964a Compare April 26, 2018 07:35

tlrx requested a review from ywelsch April 26, 2018 07:35

ywelsch suggested changes Apr 26, 2018

View reviewed changes

ywelsch approved these changes Apr 26, 2018

View reviewed changes

jasontedor approved these changes Apr 26, 2018

View reviewed changes

tlrx added 3 commits April 27, 2018 09:40

Apply feedback

c645ba5

Update list of plugins in gradle projects

7eeb112

tlrx force-pushed the add-qa-third-party-amazon-s3 branch from 409a480 to 7eeb112 Compare April 27, 2018 08:59

tlrx merged commit 7ae3b3b into elastic:master Apr 27, 2018

tlrx deleted the add-qa-third-party-amazon-s3 branch April 27, 2018 14:56

tlrx added the v6.4.0 label Apr 30, 2018

tlrx mentioned this pull request May 2, 2018

Create S3 repository tests that test against a real S3 bucket #20862

Closed

tlrx mentioned this pull request Jun 7, 2018

Use fixture to test repository-s3 plugin #29296

Merged

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move repository-s3 fixture tests to QA third party project #29372

Move repository-s3 fixture tests to QA third party project #29372

tlrx commented Apr 4, 2018

elasticmachine commented Apr 4, 2018

tlrx Apr 5, 2018

tlrx commented Apr 17, 2018

ywelsch left a comment

ywelsch Apr 26, 2018

tlrx Apr 26, 2018

tlrx commented Apr 26, 2018

ywelsch left a comment

ywelsch Apr 26, 2018

tlrx Apr 27, 2018

rjernst commented Apr 26, 2018

jasontedor commented Apr 26, 2018

tlrx commented Apr 27, 2018

Move repository-s3 fixture tests to QA third party project #29372

Move repository-s3 fixture tests to QA third party project #29372

Conversation

tlrx commented Apr 4, 2018

elasticmachine commented Apr 4, 2018

tlrx Apr 5, 2018

Choose a reason for hiding this comment

tlrx commented Apr 17, 2018

ywelsch left a comment

Choose a reason for hiding this comment

ywelsch Apr 26, 2018

Choose a reason for hiding this comment

tlrx Apr 26, 2018

Choose a reason for hiding this comment

tlrx commented Apr 26, 2018

ywelsch left a comment

Choose a reason for hiding this comment

ywelsch Apr 26, 2018

Choose a reason for hiding this comment

tlrx Apr 27, 2018

Choose a reason for hiding this comment

rjernst commented Apr 26, 2018

jasontedor commented Apr 26, 2018

tlrx commented Apr 27, 2018